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Abstract 

Social scientists are confronted with the task of researching 
idiosyncratic entities and constructs that exist in a variable 
universe. As a result, explaining the results of social science 
research is a daunting task. Nevertheless, all scientists hope to find 
answers to research questions that are true, and therefore replicable. 
Unfortunately, evidence of replicability is often missing from 
reported research. Developed by Efron (1979) and his colleagues 
(Diaconis & Efron, 1983) , bootstrap methods have the goal of creating 
an empirical sampling distribution that can be used to test 
statistical hypotheses, estimate standard errors, and create 
confidence intervals. Bootstrapping methods offer a unique and 
effective method for testing the stability and replicability of 
results. This paper explains the bootstrap method of exploring 
replicability internally, including a heuristic example applying 
bootstrap methods to a confirmatory factor analysis, using SPSS and 
AMOS . 
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Using Bootstrap Methods with Popular Statistical Packages 

Social scientists are confronted with the task of researching 
idiosyncratic entities and constructs that exist in a variable 
universe. As a result, explaining the results of social science 
research is a daunting task. Often, there are several plausible 
explanations for the results that have been obtained. Any single study 
is limited in its capacity to explain the complex phenomena that are 
usually the subject of social science research. Nevertheless, all 
scientists hope to find answers to research questions that are true, 
and therefore replicable. Thompson (1995) noted, "the most promising 
strategies emphasize interpretation based on the estimated likelihood 
that results will replicate" (p. 86) . 

Unfortunately, evidence of replicability is often missing from 
reported research. This is in no small part due to researchers' 
misconceptions that statistical significance tests to provide evidence 
of both importance and replicability. In fact, statistical 
significance tests do neither (Cohen, 1994; Thompson, 1994b, 1995). 
This topic has received much attention in recent methodology 
literature and remains controversial. The APA Task Force on 
Statistical Inference (Wilkinson & APA Task Force, 1999) recommended 
requiring researchers to address the stability of their results by 
methods such as effect size reporting (e.g., R^, eta^) and comparing 
results to previous studies. 

The only true way to know the replicability of a study is to draw 
a new sample and do the study again. This is called "external" 
replication and is rarely, if ever, done in the social sciences. Other 
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methods of exploring replicability "internally" are available. Rodgers 
(1999) provides a very accessible overview of the bootstrap, the 
jackknife, and the randomization test. The present paper will explain 
the bootstrap method of exploring replicability internally, including 
a heuristic example applying bootstrap methods to a confirmatory 
factor analysis, using SPSS and AMOS. 

Understanding the Sampling Distribution 

Before explaining the particulars of the bootstrap method, a 
brief review of the different types of distributions used in research 
is prudent. There are three types of distributions that one needs to 
understand in order to understand the role of the bootstrap: the 
population distribution, the sample distribution, and the sampling 
distribution . 

Picture a histogram of scores , with one asterisk for each person 
in the population of interest. This is the population distribution. 
Calculations that characterize the population (e.g., means, variances, 
ranges, correlations) are called parameters. However, the entire 
population of interest is rarely available for study, because most 
researchers want to generalize their results to the widest possible 
population. Therefore, the researcher draws a sample of the 
population. 

The histogram of scores for this sub-group of the population is 
the sample distribution. Statistics are calculations that characterize 
a sample. No two samples will be exactly the same. The uniqueness that 
characterizes a sample is called sampling error. Statistical 
significance tests evaluate the probability that the difference 
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between statistics is due to sampling error. [An important point to 
remember is that, as Cohen (1994) pointed out, the population used in 
statistical significance testing is presumed to be one in which the 
null hypothesis is perfectly true (e . g . , no difference between groups, 
no treatment effect).] The estimation of this probability invokes the 
sampling distribution . 

Although both the population and the sample consist of scores , 
the sampling distribution consists instead of sample statistics 
estimating population parameters. [A hybrid special case arises when 
(a) the statistic of interest is the mean and (b) n=l, because in this 
case each sample statistic in the sampling distribution is also a 
statistic from the population.] So, we must specify what parameter we 
are estimating in order to carry forward this discussion. Let’s 
presume that the parameter of interest is the score variance. 

Imagine that you could take every possible sample of size n from 
the population, administer a measurement, and calculate the variance. 
Each time you calculate the variance for a sample you plot that 
variance on a histogram. When you finish calculating and plotting the 
variance for each and every possible combination of n people (or 
whatever you are measuring) , you have a sampling distribution — a 
histogram of statistics (in this case, variances) obtained from 
repeated samplings, that models the sampling error. 

Each entry for a parameter estimate that is used to create the 
sampling distribution is based on exactly the same sample size, n, 
that was the sample size of the actual sample. We must use exactly 
this sample size for every estimate employed in creating the sampling 




6 



Using Bootstrap 



6 



distribution, because we are modeling how the sampling error affects 
the parameter estimates, and the amount of sampling error is largely 
influenced by the sample size, with more sampling error occurring as n 
is smaller. 

Each sampling distribution is unique to both sample size and the 
particular statistic being estimated. For example, the sampling 
distribution of variances for samples of 25 is different than one of 
variances for samples of 26, as well as being different from a 
sampling distribution of means for samples of 25. 

There are three types of sampling distributions (Rodgers, 1999) . 
The one described above is called the "idealized" sampling 
distribution, and is made up of all possible resamples from the actual 
population. However, researchers usually do not have access to the 
entire population. If they did, they would not be using a sample. 
Therefore, science has attempted to model the idealized sampling 
distribution two ways, theoretically and empirically. 

Social scientists are no doubt familiar with statistical 
significance tests such as F and t. These tests invoke "theoretical" 
sampling distributions. These distributions are mathematically derived 
and are said to represent the idealized distribution if certain 
assumptions are met (e.g., normal distribution, equality of variance). 
An analogy to using a theoretical sampling distribution is buying 
clothes "off the rack". These clothes come in many sizes and styles, 
but they are constructed with the assumption that people of a certain 
size will all be symmetrical and have the same basic measurements of 
height and width. 
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In contrast, the second way of modeling the idealized 
distribution is to create an "empirical" sampling distribution. This 
is analogous to "haute couture", where clothes are made to your exact 
measurements and specifications. Reordering or redrawing the original 
sample derives an empirical sampling distribution from the data in 
hand (Rodgers, 1999) . There are several advantages to using an 
empirical sampling distribution. For instance, the only assumption 
required for the use of an empirical sampling distribution is that the 
researcher believes the sample to be representative of the population, 
making this technique especially useful for sample data that does not 
conform to normality assumptions. Also, theoretical sampling 
distributions are not available for many statistics of interest (e.g., 
modern statistics) . An empirical sampling distribution can be created 
for any statistic of interest to the researcher (Thompson, 1994a) . 

Sir Ronald Fisher, who developed the F distribution, was 
convinced of the superiority of the empirical sampling distribution to 
the theoretical sampling distribution. Rodgers (1999) reported some of 
the relevant history and states, "Fisher felt that a test using a 
theoretical sampling distribution was valid only to the extent that it 
matched the results that would be obtained using an empirical sampling 
distribution" (p. 442). 

Rodgers suggested that researchers* reliance on test statistics 
(and therefore theoretical sampling distributions) is only an accident 
of time. Had computers been available to assist in creating the 
empirical sampling distribution, the need for theoretical 
distributions would have disappeared. Just as the haute couture method 
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of designing clothing is incredibly expensive, empirical sampling 
distributions are computationally expensive. However, with modern 
computers, an empirical sampling distribution can be created within 
seconds at essentially no cost. 

Understanding the Bootstrap Method 
Developed by Efron (1979) and his colleagues (Diaconis & Efron, 
1983) , bootstrap methods have the goal of creating an empirical 
sampling distribution that can be used to test statistical hypotheses, 
estimate standard errors, and create confidence intervals. Diaconis 
and Efron explained that the name "bootstrap" is derived from the old 
saying about pulling yourself up by your own bootstraps, reflecting 
"the fact that the one available sample gives rise to many others" (p. 
120) . Thompson (1995) explained the method: 

Conceptually, these methods involve copying the data set 
many times into an infinitely large "mega" data set. Then 
hundreds or thousands of different samples are drawn from 
the "mega" file and results are computed separately for 
each sample and then averaged, (p. 86) 

What actually happens is sampling with replacement. This method 
creates an environment where a person from the original sample could 
be drawn more than once in a given resample or not at all, but all 
resamples will have the same number as the original sample. The 
resulting empirical sampling distribution "informs the researcher 
regarding the extent to which results generalize across different 
types of samples" (Thompson, 1995, p. 86). 
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For our heuristic example, this paper used cognitive data on 301 
students (Holzinger & Swineford, 1939, pp. 81-91) to illustrate a 
bootstrap of a confirmatory factor analysis (CFA) . The model for the 
analysis is presented in Figure 1. The analyses were run using SPSS 
9.0 and Amos 4.0. Arbuckle (1997) recommends fixing the regression 
paths in the model, as opposed to fixing the variances. He warns that 
fixing the variances when performing bootstrap replications could 
result in inflated estimates of standard errors. 

Four separate analyses were run. First, a CFA of the model was 
performed on the entire population of 301 students, using 200 
bootstrap resamples. For purposes of comparison, we will treat the 301 
students as our population of interest , the one to which we want to 
generalize. This data will serve as our population distribution . Then, 
three separate CFA's were performed on a random sample of 75 students 
with 10, 200, and 2000 bootstrap resamples respectively. 

As mentioned previously, in the bootstrap resampling process, an 
individual may be selected more than once, or not at all, in any given 
resample. Using Amos, you can request a summary of the bootstrap 
samples. The output of this request resembles the data presented in 
the Appendix. For each bootstrap sample, a list of integers is 
displayed. The list should be read from left to right, beginning with 
the first row. The first integer tells how often the first person in 
the original sample appeared in the bootstrap sample. The second 
integer tells how often the second person appeared, and so on. 

The careful reader will notice that "Bootstrap Sample 2" and 
"Bootstrap Sample 6" appear twice, with different numbers in each set. 
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This is because Amos was unable to find a solution to two of the 
samples, thus drawing two more, for a total of twelve resamples drawn 
to obtain 10 usable samples . This information can be used to compute 
the average number of times people were used in the resamples, which 
should very nearly equal the number of resamples if the process was 
truly random. 

Using the bootstrap methods with multivariate techniques, such as 
CFA, requires some special considerations. Thompson (1994a, 1995) 
provides a thorough explanation of this topic. Because factors tend to 
fluctuate in their orders over samples. Factor I might reflect Speed 
in one sample and Memory in the next. As Thompson (1995) wryly noted: 

If the analyst computed the mean structure (or 
pattern) coefficients for the first variable on the 
first component across all the repeated samplings, the 
mean would be a nonsensical mess representing an 
average of some apples, some oranges, and perhaps some 
kiwi. (pp. 88“89) 

To provide control, Procrustean methods are used to rotate all factors 
into a common factor space. Declaring a target matrix can do this. A 
target matrix might be made up of (a) ones, zeroes, and negatives 
one’s modeling a simple structure based on theoretical expectations, 
(b) a structure or weight matrix from previous research or the data in 
hand, or (c) a graphic model such as ours, similar to those used in 
structural equation modeling. 
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Exploring Bootstrap Results in Amos 

In the text output, following the Maximum Likelihood Estimates, 
Amos prints out the Bootstrap Results. First, is a Summary of 
Bootstrap Iterations, followed by the Summary of Bootstrap Samples (as 
in the Appendix) , if this has been requested. Then, Amos provides the 
Bootstrap Standard Errors. This section presents a summary of the 
statistics from the empirical sampling distributions that were created 
by the bootstrapping process. Table 1 summarizes some of these results 
for the population. For example, the beta (B) weight (pattern 
coefficient) for the Speeded Addition Test (spdadd) reported is the 
mean value from the empirical sampling distribution of 200 pattern 
coefficients over 200 bootstrap samples. Table 1 reports statistics 
from 12 different empirical sampling distributions (one for each 
variable, for each statistic) . All in all, 28 separate empirical 
sampling distributions were created by Amos to estimate the reported 
bootstrap results, each consisting of 200 estimates. 

The standard error (SE) reported is the standard deviation of the 
empirical sampling distribution. The SE is an extremely important 
statistic in all inferential analyses (Thompson, 1994a) . The ratio of 
a statistic to the SE of a theoretical sampling distribution is 
variously known as t, F, critical ratio, and Wald*s statistic. 
Bootstrap estimates of SE can also be used for inferential purposes, 
but the have an added benefit that theoretical SE * s do not offer. 
Bootstrap SE estimates give the researcher an idea of the stability of 
the estimates over hundreds (or thousands) of configurations of the 
sample population. In spite of commonly held beliefs and practices, it 
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is this descriptive use of the SE that speaks to replicability, not 
the inferential use described above. 

If the sample statistic is relatively equal to the mean bootstap 
estimate, and the SE is small in relation to the mean bootstrap 
estimate, then the sample statistic can be thought of as stable and, 
therefore, more likely to replicate. On the other hand, if the SE is 
large in relation to the mean bootstrap estimate or the difference 
between the mean bootstrap estimate and the sample statistic reflects 
a great deal of bias, then caution interpreting the sample data is 
warranted, as the results show instability. To further assist the 
researcher with this task, Amos provides Bias estimates and the SE of 
the SE along with the other bootstrap estimates. 

Because maximum likelihood estimates (the default in Amos) assume 
normality in the distribution of the sample, a noticeable difference 
in the original estimate and the bootstrap estimates might indicate a 
deviation of the sample distribution from normal. Table 2 gives an 
overview of the descriptive statistics for the population and the 
sample. A large SE could indicate the presence of problems in the 
distribution due to sampling error. For instance. Tables 3, 4, and 5 
present the original and bootstrapped estimates for the pattern 
coefficients, the squared multiple correlations, and the error 
variances for comparison. Even the most cursory perusal will draw the 
eye to the measure variable "memnumb". Bootstrap estimates of all of 
the statistics for "memnumb" are anomalous in relation to the other 
variables. The results themselves seem impossible, including negative 
error variances and estimates of over 400. Explaining the source of 
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these anomalies is beyond the scope of this paper (and this writer) . 
However, clearly there is a problem with this variable that needs to 
be examined. More important, the problem is not apparent in the 
original estimates, only in the bootstrapped estimates. 

Because, in this heuristic example, we have the luxury of knowing 
our population parameters, we can see how well our sample did at 
estimating the characteristics present in the population. A comparison 
of Table 1 and Table 2 illustrates that while the sample did a 
moderately good job at representing the parameters for the Speed 
factor, the results are quite different for the Memory factor. 
However, pattern coefficients are often notoriously unstable across 
changes in the sample (Thompson, 1994b) . Yet, comparing Table 1 with 
Table 3, we see that "spdadd", "spdcount", "memnumb", and "memshape" 
all vary noticeably from the population parameters. Using the sample 
estimates alone, a researcher might easily have drawn erroneous 
conclusions about the population. 

In fact, we do not need to compare to Table 1. By investigating 
the estimates and the SB's for these variables, we notice that the 
SE estimates are nearly equal to or greater than the estimates on 
those variables. This is indicative of instability. Bootstrapping 
methods are the only way to get an estimate of standard error for 
multiple correlations, as Amos does not provide them otherwise. 

Another use of the empirical sampling distribution that speaks 
both to replicability and importance is the creation of confidence 
intervals. The earlier mentioned APA Task Force on Statistical 
Inference (Wilkinson & APA Task Force, 1999) , also strongly 
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recommended reporting of confidence intervals to address the stability 
of the results obtained and to give the reader a context for 
interpretation of results. These suggestions will likely become 
requirements for all APA journals in the near future. Several (i.e., 
13) journals are already requiring the reporting of effect sizes 
(Vacha-Haase, Kogan, & Thompson, 2000) . If requested, Amos will 
provide confidence intervals for any statistic that has a bootstrap 
estimate . 

Advantages and Limitations 

Bootstrapping is a unique and effective method for testing the 
stability and replicability of results. While the bootstrap results of 
our sample data caused concern about the stability of our results, the 
same method could strengthen readers’ (and editors’) confidence in 
results that are found to be stable. Remember, the original Holzinger 
and Swine ford (1939) study used the entire sample of 301 students . The 
bootstrapped SE estimates on that study (after 200 resamples) range 
from .05“. 09! 

The advantages of using bootstrap methods include not having to 
conform to distributional assumptions, or worry that violating 
assumptions somehow affected the outcome of the analysis. This paper 
presented one example applying bootstrap methods to a CFA with one 
model. Bootstrap methods can also be used to compare estimation 
methods and competing models (Arbuckle, 1997) . Beyond CFA, these 
methods can be used with any analysis (e.g., t-test, ANOVA, 
Correlations, Canonical Correlation Analysis, etc.) and with any 
statistic (e.g., Roy’s largest root, Wilk’s lambda, trimmed means) . 
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There are limits to the useful application of bootstrapping 
methods, however. Bootstrapping should be done only on relatively 
large samples. As a result, Thompson (1994a) recommended using samples 
of 40 or larger. Another important limitation is that bootstrapping 
cannot surpass or correct the limits of the data or the design of a 
study. Similarly, if you happened to pull a sample of people with an 
of .99, you will never get a large standard error, even if the actual 
population parameter is quite different. Lunneborg (1999) cautions 
researchers to incorporate knowledge of the design into the 
interpretations of bootstrap estimates. Bootstrapping cannot turn a 
descriptive design into an inferential one. Conclusions drawn from 
bootstrapped results should match the design of the study. 

Bootstrap methods are computationally complex, yet the desktop 
computer used to perform this study computed 2000 bootstrap samples in 
less that three minutes. More and more computer software is becoming 
available, as well. Amos can work in conjunction with SPSS. Thompson 
(1994a) and Rodgers (1999) mention several other programs that 
bootstrap as well, including EQS, S Plus, Resampling Stats, and SAS 
(by downloading macro programs from their website) . 

As Thompson (1996) insisted, "If science is the business of 
discovering replicable effects, because statistical significance tests 
do no evaluate result replicability, then researchers should use and 
report some strategies that ^ evaluate the replicability of their 
results" (p. 29) . There seems to be little in the way of empirically 
exploring the internal replicability of obtained results. Increasing 
awareness that statistical significance tests do not address 
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replicability will necessitate researchers finding other ways to 
attend to this important issue. 
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Table 1 



Summary of Bootstrapped ^Population” Parameters 







Parameter 


Estimates 




Factor and 
Variables 


B 


SE 




SE 


Speed 


spdadd 


.715 


.05 


.514 


.08 


spdcode 


. 647 


.07 


. 423 


.09 


spdcount 


. 644 


.05 


.417 


.09 


Memory 


memword 


. 652 


.06 


. 428 


.08 


memnumb 


.581 


.06 


.341 


.07 


memshape 


. 608 


.07 


. 374 


.09 




21 



Using Bootstrap 21 



Table 2 

Descriptive Statistics of ”Population” and Sample Distributions 









Measured 


Variables 






Statistics 


spdadd 


spdcode 


spdcount 


rtiemword 


memnumb 


memshape 


Population 

(n=301) 


Mean 


96.30 


69.20 


110.50 


175.20 


90.00 


102.50 


SD 


25.06 


15.67 


20.25 


11.51 


7.73 


7.63 


Skewness 


.25 


.26 


.53 


.85 


-.09 


-.95 


Kurtosis 


-.27 


.49 


1.24 


1.91 


-.17 


3.55 


Sample 

(n=75) 


Mean 


95.40 


69.24 


110.47 


175.63 


89.08 


101.65 


SD 


25.67 


14.99 


20.71 


9.68 


7.74 


8.23 


Skewness 


-.09 


.20 


1.14 


-.25 


. 01 


-2.11 


Kurtosis 


-1.07 


.71 


3.66 


.38 


-.19 


9.78 


Note. The 


skewness and 


kurtosis are 


used in 


inferential 


applications 


of the 


bootstrap 


results. The 


skewness and 


kurtosis 


are used in 


determining 


the 



value of the alpha percentage. 
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Table 3 

Sununary of Original and Bootstrapped Estimates of Standardized Regression 
Weights (Pattern Coefficients) for Sample of 75 



Paths 




Bootstrapped 


Estimates 


by No. 


of Resamples 


10 




200 




2000'^ 




B^ 


SE‘ 


B^ 


SE^ 


B^ 


SE'^ 


spdadd^Speed 


.776 


. 781 


.24 


.834 


.34 


.839 


.29 


spdcode^Speed 


. 649 


.710 


.28 


. 643 


. 18 


. 644 


.18 


spdcount^Speed 


.593 


.526 


.24 


.587 


. 15 


. 577 


.15 


memword^Memory 


.551 


.507 


.19 


.554 


.21 


.544 


.22 


memnumb ^Memory 


. 626 


.815 


.54 


1.808 


11.19 


3.541 20 


,03 


memshape ^Memory 


.383 


.403 


.17 


.386 


.165 


.375 


.18 


^Original estimates 


without 


bootstrapping . 


Amos 


does not 


provide 


standard 





error estimates. ^Reflect the mean value of sampling distribution for the 
statistic, ^The standard error is the standard deviation of the sampling 
distribution. It reflects the stability of the estimate over resamples. "^If 
using bootstrap for inferential purposes, large numbers of resamples are 
necessary. 
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Table 4 

Summary of Original and Bootstrapped Estimates of Squared Multiple 
Correlations (Communality Coefficients) for Sample of 75 



Factors and 
Variables 




Bootstrapped Estimates by No. 


of Resamples 


10 




200 


2000 






SE 


r2 


SE 


r2 


SE 


Speed 
















Spdadd 


. 602 


. 662 


.34 


.815 


1 .41 


.788 


. 92 


Spdcode 


. 421 


. 576 


.57 


.446 


.28 


.447 


.23 


Spdcount 


. 351 


.328 


.23 


.368 


.20 


.355 


.20 


Memory 
















Memword 


.304 


.289 


.18 


.352 


. 34 


. 357 


.37 


memnumb 


.392 


.929 


1.48 


a 


b 


c 


d 


memshape 


.147 


.188 


. 12 


. 176 


. 13 


. 172 


.15 


"R^=127.80 “'SE=1297.93 


‘=R^=413. 68 


'^SE=3310. 83 
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Table 5 

Summary of Original and Bootstrapped Estimates of Error Variance Estimates 
for Sample of 75 

Original Estimates Bootstrapped Estimates 

200 Resamples 2000 Resamples 

Factors Total Error Error Error 



and 

Variables 


Variance 


Variance 


SE 


Variance 


SE 


Variance 


SE 


Speed 


spdadd 


658.94 


258.52 


104 . 74 


115.54 


902.17 


134 . 68 


587.96 


spdcode 


224 . 67 


128.24 


31.18 


120.16 


47.09 


119.31 


46.72 


spdcount 


428.93 


274 .47 


58.23 


261.66 


80.33 


267 .47 


74 . 54 


Memory 


memword 


93.67 


64 . 37 


18 . 60 


58 . 64 


40.25 


58.23 


32.08 


memnumb 


59.89 


35.91 


13.88 


a 


b 


c 


d 


memshape 


67 . 77 


57.07 


10.95 


53.37 


19.13 


55.58 


21.18 



^Error Variance=-7 198 . 24 . *^SE=7 4 091 . 50 . ^Error Variance=-2 . 54e + 004 . 
"^SE=2. 06e+005. 
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